Abstract
Funding agencies (grantmakers) seeking to maximize the impact of science funding entrust experts (peer reviewers) to evaluate and select research proposals on their behalf. A standard practice in peer review—also referred to as “deferral to expertise”—is to appoint reviewers with content (topical) expertise, i.e. a background as close as possible to the research proposals evaluated, such as conducting research and publishing scientific papers in the same area of study. This practice is not backed by the general findings of the forecasting literature, which indicate no or weak advantages of topical expertise on forecasting.
However, in real-world settings such as geopolitics or financial markets, topical expertise is generally difficult to assess. By contrast, in biomedical research, where reviewers produce many publications per year, LLM-based techniques can be used to characterize the knowledge background of each expert.
We are conducting a large RCT study in the biomedical field to test the causal impact of content expertise on forecasting accuracy of research project outcomes.
Our research question is straightforward: are reviewers with strong content expertise providing more accurate predictions than reviewers in the same field but with weak content expertise?
The RCT study has received IRB approval from the NBER Review Board (FWA #00003692; IRB Ref#25_212 of December 4, 2025).
The RCT study was pre-registered on the AEA RCT Registry (AEARCTR-0017412; December 26, 2025).
The RCT was launched recently and is underway until mid-March 2026.
We ask you to predict the key and supplemental outcomes of our RCT.
Study ID sspp-2026-0002-v1
Authors
Chiara Franzoni, Andola Stanaj
Discipline
Economics, Management
Field
Other
Completion Time
10 Minutes
Close Date (UTC)
March 15, 2026
Citation
Franzoni, Chiara, and Andola Stanaj. 2026. "Causal impact of content expertise on predictions of biomedical research outcomes." Social Science Prediction Platform. February 5.
https://socialscienceprediction.org/s/yfa2d6
Incentive Details
Incentive Type
Based on forecast accuracy
Calculation Method
Discrete: Fixed payment if forecast is within bounds
Recipient Pool
Everyone receives the incentives